feat(org-spec-audit): one question, one issue — the ledger becomes the artifact - #142
Conversation
…e artifact The sweep filed a dated `<date>.md` in a draft PR every morning. Two sweeps two days apart asked three of the same questions twice — two of them under a byte-identical `maintenance-key` (org#147, org#148). An identical key that re-proposes rules out the obvious cause and leaves the real one: **the ledger could only remember "no", never "asked".** Both suppression rules key off a TERMINATED proposal — declined in the ledger, or a PR closed unmerged — and the first PR was neither. It was open, unanswered, in the queue, which the design read as no signal at all. The unstable key was the second cause and would have fixed one duplicate of the three. So the unit is the question and the artifact is a GitHub issue. open = asked (write nothing), closed = decided (never re-file, never reopen), absent = new. - `createIssue` on the github-app surface, `openIssue` on `GithubService`. It FAILS rather than degrading to a logged no-op, unlike `openDraftPullRequest`: a PR write that no-ops leaves its content in git and its branch idempotent, while a no-op here drops the only copy of the question. - `listIssues` gains `strict`, which throws instead of returning a list the page ceiling cut short. A dedup read cannot use a partial list — it answers "not filed" for everything it never reached — and a bare array cannot distinguish "ended on a short page" from "ended on the ceiling". - `IssueRef.closedAt` — the column `pulls` lacks and `issues` has. - Reconciliation: exact key match first, then ONE model call over the residue asking "one of these, or new?". Every answer is checked against the keys actually read from the control repo, so a hallucinated match files the question rather than silently burying it. The prompt now asks for noun-phrase keys (`pipeline-dags`, not `adopt-pipeline-dags`). - The dedup read happens BEFORE the sweep and fails CLOSED, inverting the suppression primitive's posture: an unreadable ledger cost one duplicate PR, an unreadable issue set costs a duplicate of every question at once. The declines ledger still fails open — its failure costs one issue a human closes, and that close then suppresses it for good. - `checkSuppression`'s `headBranchPrefix` is optional; omitted, the PR-history read is skipped entirely rather than made and ignored. - The notice is a delta: filed today with links, standing open count, what the per-sweep cap held. `max-new-questions` bounds the writes; the old per-group rendering cap is gone. - The github fake appends every `openIssue` to the list `issues` reads back, so "the second sweep sees the first sweep's issue" is testable at all. Config: `questions-dir` retires for `questions-label`, `lane-label-prefix` and `max-new-questions`. A label that is set-and-unusable fails the run instead of falling back — it is both the filter the read applies and the label the write applies, and a comma makes those two different things silently. Design: fractalboxdev/org#150.
There was a problem hiding this comment.
AI code review — 💬 Comment
Risk tier: full · 0 critical · 2 warnings · 0 suggestions
Reviewers: security 0 · performance 1 · code-quality
1. ⚠️ Warning — Strict issue reads fail permanently once the control repository exceeds 500 issues
📍 packages/github-app/src/issues.ts:255-263
'strict' uses 'MAX_PAGES_DEFAULT' (5) when the caller does not provide 'maxPages', then throws as soon as the fifth page is full. The spec-audit deduplication read enables 'strict' but does not configure a larger or adaptive page limit, so a control repository with more than 500 issues makes every sweep fail after five sequential API calls rather than completing. This will become a recurring availability/performance problem as the ledger grows; use an unbounded/adaptive strict pagination strategy or a targeted server-side query for the maintenance key.
2. ⚠️ Warning — Documented URL invariant is not enforced
📍 packages/core/src/services/github.ts:68-69
The 'IssueCreated.url' documentation says the URL is never empty on success, but 'createIssue' only validates the issue number and returns 'str(created.html_url)', which can be '""' when 'html_url' is missing or malformed. Either validate the URL and fail the create, or weaken the documentation to describe the possible empty value.
…e needs its url Two findings from pr-review, both real. `strict` inherited the library's 5-page default, so once the control repo held 500 label-filtered issues EVERY sweep would fail — permanently, five API calls in. The ledger read now asks for 20 pages, and the error names the remedy. Read the number as a canary: 2,000 questions asked and never closed is not a pagination problem, it is the loop having outrun the team, and a red run there beats one deciding on the first 500. It costs nothing until then — pagination stops on the first short page. `IssueCreated.url` was documented as never-empty and wasn't. A caller announces the issue by linking it, so `createIssue` now fails on a missing `html_url` the same way it fails on a missing number, rather than publishing a link to nowhere. The fake records `maxPages` and the run test asserts the raised ceiling, so a later edit dropping it cannot silently restore the five-page failure.
|
Both fixed in 5754a9d. 1. Strict reads and the 5-page default — correct, and the failure is worse than described: it is permanent. Once the label-filtered set exceeds 500, every sweep fails five API calls in, forever, until someone raises the number. The ledger read now passes I did not make it unbounded or adaptive, and the reasoning is worth stating: this read's value is completeness, so the only alternatives are "paginate until exhausted" (unbounded work on a scheduled job) or "decide on a partial list" (the bug the flag exists to prevent). A high explicit ceiling keeps the third option — fail loudly — and costs nothing until reached, since pagination stops on the first short page. 2,000 questions asked and never closed is not a pagination problem; it is the loop having outrun the team, which the process doc names as this design's actual failure mode. A red run there is closer to correct than a silent partial read. Search-by-key was the other candidate and I rejected it on indexing lag: 2. Also pinned both: the fake records |
org-spec-auditfiles one GitHub issue per open question in the control repo instead of a dated markdown file in a daily draft PR. A sweep that re-finds a question already on file writes nothing — no comment, no reopen, no second issue. Implements the design in fractalboxdev/org#150.Problem & Insight
The sweep produced org#147 and org#148 two days apart, four questions each, three of them the same question twice — two under a byte-identical
maintenance-key.An identical key that re-proposes rules out the obvious cause and leaves the real one. The ledger could only remember "no". It had no way to remember "asked". Both
checkSuppressionrules key off a terminated proposal — a decline indeclined.jsonl, or a PR closed unmerged — and the first PR was neither. It was open, unanswered, in the queue, which the design read as no signal at all, so the second sweep re-proposed a question whose first proposal was still on screen, exactly as specified.The unstable key is the second cause, not the first: the same question minted
authorize-pipeline-dagsone day andadopt-pipeline-dagsthe next. A verb encodes the proposed action, which is phrasing; the subject is not. Fixing only that would have fixed one duplicate of three.Take
An issue already is what the suppression machinery was approximating, and it holds the state a ledger of declines structurally cannot:
openanswers "have I raised this?". One field, three meanings, no second file.open→ write nothing;closed→ never re-file, never reopen;absent→ file.Decisions worth review:
openIssuefails rather than degrading to a logged no-op, unlikeopenDraftPullRequest. A PR write that no-ops loses nothing — idempotent branch, content in git — while a no-op here drops the only copy of the question.listIssuesgainsstrict. A dedup read cannot use a list the page ceiling cut short — it answers "not filed" for everything it never reached — and a bare array can't distinguish "ended on a short page" from "ended on the ceiling". Opt-in, so no triage caller changes.checkSuppression'sheadBranchPrefixis now optional. Omitted, the PR-history read is skipped entirely rather than made and ignored: a prefix matching nothing would answer "no prior proposals" every tick for a reason no reader could tell from the feature being off.The
githubfake appends everyopenIssueto the listissuesreads back — without that, every test would pass against a run that dedups against nothing.Config:
questions-dir→questions-label/lane-label-prefix/max-new-questions. A set-and-unusable label fails the run rather than falling back; it is both the filter the read applies and the label the write applies, and a comma makes those two different things with nothing erroring.Verification
pnpm typecheckclean,pnpm lintclean,oxfmt --checkclean on all 13 changed files. 60 run tests + 24 github-app issue tests pass; full suite 2236 passed. The 5 failing files inpackages/demo-agentare missing optional deps (@effect/ai,puppeteer-core,pngjs) and fail identically on a clean tree.Follow-ups
infra/maintenance-loop/sync-config.sh --applyinfractalboxdev/orgafter this deploysorg-spec-audit.questions-dir; deleting before this ships would silently move the daily file to the defaultmaintenance/questionsdeclined.jsonlentries